Performance assessment of primary health care facilities in Brazil: Concordance between web-based questionnaire and in-person interviews with health personnel

This study is a concordance analysis comparing answers to two external assessment tools for Primary Health Care (PHC) facilities that use two different data collection methodologies: (a) external assessment through structured interviews and direct observation of facilities conducted by the National Program for Improvement of Access and Quality of Primary Care (AE-PMAQ-AB), and (b) a computerized web-based self-administered questionnaire for Assessment of the Quality of Primary Health Care Services (QualiAB). The two surveys were answered by 1,898 facilities located in 437 municipalities in the state of São Paulo, Brazil, between 2017 and 2018. Both surveys aimed to assess the management and organization of PHC facilities. A total of 158 equivalent questions were identified. The answers were grouped by thematic similarity into nine domains: Territory characteristics; Local management and external support; Structure; Health promotion, disease prevention, and therapeutic procedures; Attention to unscheduled patients; Women’s health; Children’s health; Attention to chronic conditions; and Oral health. The results show a high level of concordance between the answers, with 81% of the 158 compared questions showing concordance higher than 0.700. We showed that the information obtained by the web-based survey QualiAB was comparable to that of the structured interview-based AE-PMAQ-AB, which is considered the gold standard. This is important because web-based surveys are more practical and convenient, and do not require trained interviewers. Online assessment surveys can allow immediate access to answers, reports and guidelines for each evaluated facility, as provided by the QualiAB system. In this way, the answers to this type of survey can be directly employed by users, allowing the assessment to fulfill all phases of an assessment process.

Introduction criteria and standards following the PNAB and SUS guidelines; adherence is voluntary and not linked to financial incentives. Eight QualiAB surveys have been conducted in different regions of the country. In the state of São Paulo, the last survey was conducted in 2017, preceding the AE-PMAQ-AB in the state by a few months [17].
By focusing on a particular fraction of the complex aspects involved in health facility assessment, this study investigates whether there is concordance between the responses to two assessment surveys that used structured questionnaires and adopted different data collection methodologies: the PMAQ-AB external assessment (AE-PMAQ-AB), conducted by in-person interviewers and considered here as the "gold standard," and the QualiAB external assessment, conducted through a computerized web-based self-response system.

Materials and methods
This is a concordance analysis study comparing answers of 1,855 PHC facilities to two surveys conducted with the aid of structured instruments in the state of São Paulo, Brazil, in 2017Brazil, in -2018. The two surveys consist in external assessments that use different information collection methods: the AE-PMAQ-AB assessment is conducted by in-person interviewers, whereas the QualiAB assessment uses a web-based self-response questionnaire.
The selection of assessment surveys was based on the comparability between answers, according to the equivalence of the following criteria: 1) focus on the organization of the work process; 2) use of the same technical and political references to define the criteria and standards of their indicators; 3) use of structure and process indicators; 4) occurrence in periods close to each other; and 5) assessment of PHC facilities in the same region.
The assessments were carried out at PHC facilities in the state of São Paulo, which has a municipally managed network of public PHC facilities covering approximately 60% of the population during the period studied. The state has 45.5 million inhabitants (21.9% of the Brazilian population) and has the country's second highest Human Development Index (HDI): 0.783. However, its 645 municipalities display great geographical, populational, and socioeconomic heterogeneity. Forty percent of the municipalities have fewer than 10,000 inhabitants, are geographically distributed across coastal and mountainous regions that are poorly accessible, and have municipal HDIs between 0.862 and 0.639, thus presenting inequalities similar to Brazil's [18][19][20].
The AE-PMAQ-AB assessment under analysis was conducted in the state of São Paulo between May and August 2018, as part of the third PMAQ-AB cycle. It was the result of a partnership between the Ministry of Health and public higher education institutions responsible for selecting, training, and hiring the university-level professionals who conducted the AE-P-MAQ-AB data collection in loco16. The interviewers formed regionalized teams supervised by a coordinator responsible for planning their travel itinerary and checking the recorded information. Data collection was carried out through structured questionnaires installed on tablets equipped with a computerized system that sends the data to the Ministry of Health at the end of each interview. In the state of São Paulo, 2,693 family health teams based in 564 municipalities were assessed in loco. The municipal participation rate was 87.4%. No data are available on the total number of PHC facilities in the municipalities participating in the PMAQ assessment. Participation was encouraged by the Ministry of Health and partner institutions through presentations of the project in regional meetings and financial incentives to participate [21]. The Ministry of Health was slow in disclosing the end results of the PMAQ assessment. The final score determined the ranking of each facility and the amount of financial incentive for performance. The score was composed of the results achieved in three stages: (1) implementation of self-assessment procedures, accounting for 10% of the score; (2) assessment of contractual indicators, corresponding to 30%; and (3) the AE-PMAQ-AB assessment, which was the last stage and accounted for 60% of the final score [21]. The partial results relative to the AE-PMA-Q-AB stage were not disclosed to the participants.
The QualiAB survey was conducted in the state of São Paulo between May and November 2017 as part of a research project resulting from the partnership of two public higher education institutions (UNESP and USP) and the São Paulo State Department of Health (SES SP). It was supported by the Council of Municipal Health Secretaries of the state of São Paulo (COSEMS SP) and encompassed 2,739 PHC facilities located in 514 municipalities in the state of São Paulo. A total of 79.7% of municipalities participated in the QualiAB survey, with a participation rate of 88.2% for the PHC facilities located in the participating municipalities. The Qua-liAB assessment project was presented by the SES SP to municipal health secretaries in regional meetings to encourage municipal participation. The other stages of the assessment were carried out via web: enrollment of the municipalities with password definition; enrollment of facilities by local managers with individualized access passwords; questionnaire response; and hierarchized access to results and standards according to the institutional rank of the participants. The computerized system provided participants with immediate access to their score and performance level, measured both globally and by indicator, as well as access to recommendations according to the criteria and standards used. There was no financial incentive to participate in the QualiAB assessment [22].
Both questionnaires were written in Portuguese, Brazil's official language, and answered by two different professionals of each PHC facility. The content of the questionnaires used in both assessments was analyzed at two different times by two researchers, who identified 158 comparable questions. Even though these questions are not identical, they have equivalent phrasings, as they address the same aspects of work organization. The QualiAB answer alternatives tend to be more detailed, while in the AE-PMAQ-AB assessment, they are overall dichotomous (yes/no), as shown in Table 1. To allow comparison, the equivalence was established through the presence, or lack thereof, of paired items.
The 158 comparable questions were grouped into nine domains relative to territory, structure, management, and attention to main demand and programs, as provided in the Brazilian National Primary Care Policy [23]. The domains comprehend different aspects of the organization of PHC facilities and make it possible to analyze concordance differences between facilities. The nine domains are: 1. Territory characteristics; 2. Local management and external support; 3. Structure; 4. Health promotion, disease prevention, and therapeutic procedures; 5. Attention to unscheduled patients; 6. Women's health; 7. Children's health; 8. Attention to chronic conditions; 9. Oral health.
After the comparable questions were identified, the facilities that responded to the two external assessments in the state of São Paulo were paired, resulting in a total of 1,855 facilities located in 434 municipalities.
The AE-PMAQ-AB assessment was used as the gold standard to validate the answers. This choice is justified because the AE-PMAQ-AB assessment was part of PMAQ, an official program of the Brazilian government for assessing PHC facilities during the period analyzed in this study, and because its criteria, standards, and indicators reflected the technical and political proposals for PHC facilities [16,23]. That choice is also corroborated by the fact that PMAQ has nationwide reach and defines a "baseline" for PHC assessment in Brazil, which is confirmed by the multiple studies and assessments based on AE-PMAQ-AB data [14,[24][25][26][27].
The proportion of similar answers to the comparable questions from the two questionnaires was calculated, which made it possible to calculate sensitivity (TP/(TP+FN)), specificity (TN/ (FP+TN)), and accuracy ((TP+TN)/N). TP stands for "true positive" (where both questionnaires have an affirmative answer for the same comparable question); FN stands for "false negative" (affirmative answer in QualiAB, negative in AE-PMAQ-AB); TN is "true negative" (negative answers in both questionnaires); FP is "false positive" (negative answer in QualiAB, affirmative in AE-PMAQ-AB); and N is the total of answers. The concordance between answers was analyzed with the Kappa coefficient test; all analyses were performed using the SPSS

AE-PMAQ-AB structured interviews QualiAB web-based survey
Is the team's coverage area defined? The facility's coverage area is defined: Choose only one alternative □ Yes □ 1) Administratively according to the central level of the Health Secretariat or other municipal health agency □ No □ 2) Through participative planning, considering the local reality and ease of access □ 3) In practice, the team defines an area to carry out actions in the community To compare the item "Defined coverage area", AE-PMAQ-AB questions and QualiAB questions were used, which inquired about the existence or not of a defined coverage area, regardless of how it has been defined.
Vaccines at the health facility The following vaccines are administered at the facility: General-Always available hepatitis B

Results
The pairing of questions from the two external assessments made possible the analysis of the concordance of the answers from 1,855 PHC facilities, geographically distributed across 67% of the municipalities of the state of São Paulo (Fig 1). The grouping of questions into nine domains allowed for the analysis of the main aspects involved in the organization of the PHC facilities. A high concordance level was found between the answers to both assessments: 81% (128) of the QualiAB answers showed a concordance level higher than 0.700 in relation to the AE-PMAQ-AB answers. Only in the question addressing treatment of people living with HIV/AIDS was the concordance level lower than 0.500 (Acc = 0,414) ( Table 2). Table 2 comprehends domains relative to territory, some of the available resources, and management characteristics, showing high accuracy for nearly all compared questions, and varying sensitivity and specificity.
In Table 3, all the questions show accuracy higher than 0.900. While sensitivity is high for all three subdomains (health promotion, disease prevention, and therapeutic procedures), specificity is considerably lower for the subdomain Therapeutic Procedures.  Table 4 shows low accuracy for the question about risk assessment of unscheduled patients, as well as for the questions in the subdomain Attention to Chronic Communicable Conditions, except for the question about bacilloscopy for tuberculosis. In the subdomain of Chronic Noncommunicable Conditions, the questions about ocular fundus examination and mental health care also show low accuracy. The item with the lowest accuracy, "Delivery of care to persons living with HIV/AIDS", contrasts with the high accuracy seen in consultations for persons with diabetes and hypertension, as well as in the delivery of care to women and children.

PLOS ONE
The high accuracy combined with low, even negative, Kappa values is explained by the vulnerability of the Kappa coefficient test against marginal distributions and asymmetric joint distributions, since too high concordances, without a normal distribution, compromise Kappa values [27].

PLOS ONE
Concordance between web-based questionnaire and in-person interview in primary health care assessment

Discussion
The results presented a high concordance level between answers to the paired questions of both assessments, showing that web-based questionnaires are a viable tool to assess work organization in PHC facilities when it comes to structure and processes. The highest accuracy level was found in the more traditional actions of the Brazilian PHC programs, especially in relation to questions of health promotion, disease prevention, therapeutic procedures, definition of coverage area, local management, and health care programs such as care delivery to women, children, persons with hypertension and type 2 diabetes, and more traditional oral health procedures.
The lowest-accuracy questions may point to a lack of clarity in their formulation, respondents' limited knowledge of the subject, or recent implementation of the service in question. For example, the item "Dedicated vaccine refrigerator" showed low accuracy level (0.533), which may be linked to the lack of clarity in the question's phrasing. Neither the QualiAB nor the AE-PMAQ-AB questionnaire specifies the type of refrigerator (for home or commercial use), which may have led to diverging interpretations, as it is recommended to replace homeuse refrigerators with ones that meet safety and quality standards [28].
The lowest accuracy level among the 158 paired questions was found in "Delivery of care to persons living with HIV/AIDS" (0.414), which has only recently been incorporated into the Brazilian PHC [29], followed by "Ulotomy/Ulectomy" (0.516), which are low-demand procedures unknown to many team members [30]. It is important to point out that the AE-PMA-Q-AB questionnaire should be answered either by the facility manager or by the head of each department, or even by a doctor, whereas the QualiAB questionnaire should preferably be answered in a team meeting. It behooved the facility personnel to find the best way to answer Table 3. Comparison of the answers to the QualiAB and the AE-PMAQ-AB questionnaires relative to health promotion, disease prevention, and therapeutic procedures, according to accuracy, sensitivity, specificity, confidence interval, and Kappa coefficient. Brazil

PLOS ONE
Concordance between web-based questionnaire and in-person interview in primary health care assessment the questionnaires without compromising service to patients. They were, therefore, answered by different team members of each facility, which may have influenced the answers regarding low-demand or recently implemented services.
Other not fully implemented services, such as electronic medical records, risk assessment protocols for persons with diabetes and hypertension, and mental health care, also showed low accuracy levels, which may be related to the reorganization of the facilities during that period.
In general, the questions showed lower specificity, i.e., higher discordance in relation to negative answers. In addition to the possible reasons mentioned above, this discrepancy may be due to the adjustment to expected standards as a result of the self-assessment process that preceded AE-PMAQ-AB. This process was a component of both the PMAQ and the QualiAB programs.
PMAQ was the first institutional program of PHC facility assessment that covered the entire Brazilian territory, involving a large number of in loco AE-PMAQ-AB interviewers. In this process, transportation difficulties arose from the great distances between municipalities. Municipalities with large rural areas also proved difficult to reach. Thus, journeys frequently took hours or days, by different routes-air, river, or land. Roads were often precarious, and weather incidents, such as rainy seasons, blocked roads or isolated PHC facilities in the state of Amazonas [31][32][33]. Brazil's great territorial extension and geographic diversity make in loco data collection surveys difficult and costly in many respects. Additionally, the high cost of the whole process of organization, selection, training, and hiring of a large number of professionals must be taken into account when choosing the best form of data collection [7].
Information and Communications Technology (ICT) has long been gaining ground in the health sector, with the incorporation of telemedicine technologies that make it possible to expand patient service and health care professionals' training and support [34], as well as improving information record. As of 2020, with the measures to prevent the spread of Covid-19, the use of ICTs and computer equipment has been amplified in Brazil [35], which increases its potential for use in health care services and assessment processes.
Web-based structured assessments are limited by the number of questions that can be asked, which also limits the scope of the assessment and requires the selection of high-sensitivity and -specificity indicators. Additionally, this type of assessment requires great investment in establishing partnerships that will participate actively in and be committed to the assessment, thus yielding high response rates [7]. In-person assessments, on the other hand, even when based on structured questionnaires, make it possible not only to expand and diversify the subjects and interviewed professionals, but to observe the facilities directly and to use various instruments, such as semi-structured interviews with patients.
Some limitations of this study worthy of pointing out are the difference in subjects addressed by the two instruments, the time gap between the surveys, and their different levels of institutionalization and ability to induce participation. The AE-PMAQ-AB assessment was part of a financial incentive-granting evaluation program of the Brazilian Ministry of Health to improve service quality, whereas the QualiAB program was rather a self-assessment opportunity for the participating facilities. On the other hand, web-based assessments, such as the one conducted through the QualiAB system, in addition to not posing the challenges of in loco data collection, can provide immediate access to results, reports and orientation to the system's users. Quick access to results make them more likely to be used by the participants, particularly their direct users-the PHC teams-, thus allowing the assessment to complete all the stages of an assessment process.
Timely disclosure of results and good strategies for communicating them are mentioned in the literature [36,37] as factors that promote knowledge and use of the assessments to underpin political decision making, redesign measures, and allocate financial resources [38,39]. They also increase acceptance of the assessment, according to Rissi, Sager (2013) [40]. Another advantage is that this format does not interfere in the routine health care service, as it makes it possible to save the information record in case it is necessary to interrupt the response process. We can also add the ease of answering the questionnaire in a partial and scheduled way, which favors the involvement of a larger number of professionals in the discussion of the answers [7,17].
The high level of concordance found between both assessments points to advantages in the use of web-based assessment instruments. These advantages become even more pronounced in light of the need for investments to expand assessment surveys, amplify the assessment culture, navigate pandemic scenarios, and further computerize PHC services, thus highlighting the importance of investing in web-based assessments as one more tool to improve the quality of PHC services and facilities.